Clinical Speech to Text Evaluation Setting

Failures in information flow from clinical handover are the leading cause of sentinel events in the USA and associated with nearly half of all adverse events and over a tenth of preventable adverse events in Australia. Verbal clinical handover provides a good picture of the background clinical history and current state of clinical management of a group of patients cared for by a nursing team. However, all this valuable verbal information is lost after three consecutive shifts if no notes are taken during handover. When traditional notetaking by hand occurs, less than a third of data is transferred correctly after five shifts. We propose using an automated approach of cascading speech-to-text conversion, standardisation with respect to controlled thesauri, and structuring in accordance with documentation standards. This transcribes verbal handover information into written drafts for subsequent clinical review, editing, and addition to electronic health records. In this paper, we introduce the evaluation setting for this technology development in a laboratory environment. It ranks a wide range of recording devices used alone or in combination with headsets and lapel microphones based on clinicians’ preferences and their accuracy in speech-to-text conversion. The sample consists of four student nurses and four experienced academics from diverse clinical specialties and speaking styles. To simulate realistic nursing clinical handovers, twenty handover scenarios have been scripted. The subsequent evaluation in a clinical environment will address speech-to-text conversion, standardisation, and structuring with the short-listed devices in six hospitals with the sample of thirty authentic handover situations per hospital. To compare recorder-microphone combinations across all participants, professional-level recording devices are used to record each participant. The recordings are then played using professional-level speakers across all recordermicrophone combinations to achieve equivalency in voice input. Statistical accuracy in speech-to-text conversion with noise experimentation is used to determine the most accurate combination. Two speech-to-text systems are compared against transcription by hand. An eighteen-item pre-experimental survey addresses initial perceptions of using the proposed automated approach in clinical settings. This includes participants’ opinion on the improvement of clinical handover with the proposed automated approach, their understanding of the related technologies and perceived problems with the clinical application. An eleven-item post-experimental survey examines device usability with reference to the specific experimental devices. Each participant is asked to complete both surveys and participate in a one-to-one interview. All participants are videoed using the recording devices and accessing typical device functions to further examine human-device interactions for usability assessment. We are seeking additional partners to further develop and evaluate the approach and setting.